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environment  in  the  experiment  was  modeled  in  detail  from  this  MOUT  facility . 3 
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1.  Introduction 


As  robots  become  more  complex  and  autonomous,  there  are  challenges  in  how  to  keep  Soldiers 
informed  about  robot  activity  and  intent  and  how  to  maintain  overall  mission  effectiveness 
without  increasing  cognitive  workload.  Soldiers  must  be  able  to  quickly  gain  an  understanding  of 
a  robot’s  status  when  switching  from  one  task  to  another  task  (e.g.,  switching  between 
reconnaissance  or  a  search  for  improvised  explosive  devices  [IEDs]  to  area  security  or 
navigation).  Soldiers  will  also  need  predictive  capability  about  a  robot’s  intended  course  of 
action  so  that  corrective  actions  can  be  taken  prior  to  encountering  a  situation  in  which  it  could 
prove  difficult  for  the  autonomy  to  handle  without  operator  intervention.  In  this  study,  we 
investigated  the  usefulness  of  operator -knowledge-management  aids,  which  overlay  information 
about  autonomous  unmanned  ground  vehicles’  (UGVs’)  perception  and  planned  route  execution 
onto  existing  user  interfaces.  These  operator  aids  provide  information  visually  and  assist  in 
cognitive  tasks  that  involve  understanding  and  projection  of  robot  intent  to  follow  planned  travel 
paths. 

The  goal  of  this  study  was  to  continue  to  evaluate  the  extent  to  which  operator  performance  and 
perceived  workload  is  affected  by  operator  aids.  This  is  the  second  study  looking  at  operator  aids 
and  measuring  performance  data,  such  as  time  to  complete  task  and  number  of  operator 
interventions  (examples  of  task -dependent  measures  similar  to  those  suggested  by  Fong  et  al. 
2004).  Workload  was  evaluated  as  a  self-report  measure  using  the  NASA  Task  Load  Index 
(TLX)  (Hart  and  Staveland  1988).  Additionally,  operator -preference  data  were  collected  to 
evaluate  which  of  the  operator  aids  were  preferred  and  why. 

1.1  Background 

In  2001,  Jameson  described  the  future  Army’s  extensive  use  of  mobile  sensing  systems, 
unmanned  platforms,  and  decision-aiding  systems.  He  also  explained  that  information  fusion 
would  require  an  ability  to  create  and  maintain  real-time  situation  awareness  (SA)  from  all 
available  information. 

As  described  in  this  study,  operator  aids  are  one  way  to  assist  robotic  operators  in  maintaining 
SA  by  quickly  displaying  mission-relevant  information  without  taking  too  much  attention  from 
primary  tasks.  Operator  aids  are  display  tools  that  generate  information  from  data  supplied  by  the 
autonomous  robot  to  provide  an  increased  understanding  of  the  robot’s  perceived  environment 
and  current  and  planned  actions.  The  intent  of  operator  aids  is  to  reduce  the  cognitive  workload 
by  giving  an  operator  information  about  a  robot  in  a  way  that  limits  additional  user  cognitive 
processing  and,  at  the  same  time,  increases  operator  understanding.  Wickens’s  Multiple 
Resource  Theory  (1980,  1984)  suggests — and  specifically  for  tasks  with  overlapping  cognitive 
resources  (i.e.,  vision-based  tasks  such  as  reconnaissance) — that  as  workload  increases, 


1 


performance  decreases.  Therefore,  in  a  human-robot  team  reconnaissance  task  with  a  number  of 
highly  visual  components,  it  is  expected  that  as  workload  increases,  operator  performance  will 
decline. 

Previous  research  (Mitchell  et  al.  2013,  Mitchell  2008)  similarly  stated  this  expectation, 
predicting  larger  increases  in  workload  for  robot  operators  versus  nonoperators  because  of  the 
multiple,  overlapping  cognitive -resource  requirements.  We  propose  that  the  appropriate  use  of 
operator  aids  can  mitigate  some  of  the  increased  workload  and  performance  decline  by  making 
explicit  to  the  operator  some  of  the  information  from  the  robot  sensors  as  to  current  and  future 
robot  status  and  actions. 

Research  that  explored  shared  mental  models  and  their  effect  on  team  performance  (Mathieu  et 
al.  2000)  provides  a  starting  point  for  defining  contributions  that  make  human-robot  teams  more 
successful.  One  key  is  the  ability  among  teammates  to  engage  in  predictive  behaviors  based  on 
those  shared  mental  models.  Operator  aids  investigated  here  may  support  the  development  of  a 
team  environment  in  which  human  operators  can  predict  robot  behavior  based  on  the  additional 
information  provided  by  the  robot  through  the  knowledge-management  aids.  Specifically, 
operator  aids  focus  on  information  needed  in  order  to  supervise  the  robot  and  appropriately 
intervene  to  assist  it.  By  informing  the  operator  about  what  an  autonomous  robot  calculates  as  its 
best  route  and  why,  the  operator  may  anticipate  situations  that  could  be  difficult  for  the 
autonomy  to  address  successfully  without  operator  intervention.  Understanding  the  autonomous 
system’s  “intent”  would  allow  the  operator  to  make  timely  input  to  the  robotic  system  when 
needed.  It  is  believed  that  all  of  these  outcomes  will  improve  the  human-system  performance. 
Human  performance,  in  this  case,  is  defined  by  measures  such  as  the  amount  of  time  spent  in 
teleoperation  mode  and  the  number  of  times  the  operator  engages  manual-teleoperation  mode. 
The  less  frequently,  and  less  time,  the  operator  is  engaged  in  manual  control,  the  more  time  will 
be  available  to  the  operator  for  primary  operational  tasks  such  as  detecting  IEDs.  Operator  aids 
are  tools  with  which  human  workload  related  to  manual-teleoperation  control  can  be  reduced — 
allowing  more  time,  energy,  and  focus  for  other  mission-critical  tasks. 

1.2  Hypotheses 

Hypotheses  for  this  study  have  been  broken  up  into  1)  those  related  to  performance,  and  2)  those 
related  to  workload. 

1)  Two  performance-related  hypotheses  were  developed: 

Hi:  Participants  will  identify  simulated  targets  with  greater  accuracy  in  the  experimental 
conditions  than  in  the  control  condition. 

H2:  Participants  will  engage  teleoperation  control  (also  called  “teleop”)  less  often  and  for 
less  time  in  the  experimental  conditions  than  in  the  control  condition. 

2)  Two  hypotheses  were  formulated  in  relationship  to  workload: 
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FFs:  Participants  will  experience  lower  levels  of  perceived  workload  in  the  experimental 
conditions  than  in  the  control  condition. 

H4:  Participants’  feedback  will  indicate  a  preference  for  operating  a  robot  with  operator 
aids  as  opposed  to  having  no  operator  aids  available. 


2.  Methodology 


2.1  Experiment’s  Location  and  Technologies 

2.1.1  Experiment’s  Location 

The  Operator  Aid  study  took  place  at  2  locations  at  Marine  Corps  Base  Camp  Lejeune,  North 
Carolina  (Fig.  1)  as  part  of  Safe  Operations  for  Unmanned  Reconnaissance  in  Complex 
Environments  Army  Technology  Objective  (SOURCE  ATO),  a  multiorganizational  field- 
research  and  demonstration  event.  The  SOURCE  ATO  event  took  place  over  several  weeks  and 
involved  a  number  of  organizations  and  technologies.  Operator  aids  are  one  piece  of  this  large 
research  effort. 


Fig.  1  Aerial  view  of  the  Camp  Lejeune’s  MOUT  facility: 

Soldiers  participated  in  this  experiment  in  both  a 
classroom  and  a  stationary  Stryker  vehicle.  The  simulated 
environment  in  the  experiment  was  modeled  in  detail  from 
this  MOUT  facility. 

The  2  locations  were  an  indoor  classroom  in  the  military  operations  in  urban  terrain  (MOUT) 
facility  for  training  and  at  an  operator  station  located  in  the  back  of  a  specially  equipped  Stryker 
vehicle  called  the  Crew-integration  and  Automation  Test  bed  (CAT).  The  participants  were 
seated  in  the  vehicle  commander’s  position  within  the  CAT  (Fig.  2). 
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Fig.  2  Basic  layout  of  control  stations  in  the  CAT  Stryker: 

Soldier  participants  sat  in  the  vehicle  commander’s 
station,  front  right.  The  work  station  included  two 
17-inch  computer  screens  showing  the  simulated  robot 
operations  in  Camp  Lejeune’s  MOUT  environment. 

The  Camp  Lejeune  MOUT  facility  was  modeled  as  the  simulation  environment  used  for  the 
robotic  operations  within  this  experiment. 

2.1.2  Experiment’s  Technologies 

This  study  used  the  ModSim  simulation  software  developed  by  General  Dynamics  Robotics 
System  (GDRS).  This  simulation  was  created  as  a  visualization  platform  that  serves  as  a  basis  for 
realistic  simulated  experimentation,  validation,  and  refinement  of  hardware -in-the -loop  dynamic 
planners.  The  virtual,  3 -dimensional  (3D)  environment  simulated  was  the  streets  and  buildings  of 
the  Camp  Lejeune  MOUT  facility.  A  simulated  robot,  called  the  Unmanned  Ground  Vehicle 
(UGV),  operated  within  this  environment.  The  simulation  was  used  in  conjunction  with  the  US 
Army  Tank  Automotive  Research,  Development,  and  Engineering  Center’s  Warfighter  Machine 
Interface  (WMI),  developed  by  DCS  Corp.  The  WMI  is  a  user-interface  system  designed  to 
accommodate  a  number  of  different  technologies  through  the  use  of  various  computer-screen 
tabs  to  access  the  technologies.  The  operator  aids  are  Travel  Planner  (TP),  Obstacle  Map  (OM), 
and  Rerouting  Alert  (RA).  Examples  of  this  WMI  interface  can  be  seen  in  Figs.  3-7. 
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Fig.  3  This  example  of  the  WMI  interface  shows  2  of  the  operator 
aids  in  the  forward-facing  (3D)  view:  TP  (green  line)  and 
OM  (red/green  squares).  The  Operator  Aids  software  was 
used  while  in  the  UGV  technology  tab  (across  the  top). 
Other  control  capabilities  were  available  on  the  touch¬ 
screen  buttons  located  on  the  right  and  left  of  the  screen. 


Fig.  4  This  example  of  the  WMI  shows  the  TP  (blue  line)  operator  aid  in  the  forward-facing  (3D)  view.  The 

line  is  shown  as  completely  blue,  meaning  the  vehicle  has  stopped  (also  indicated  by  the  speedometer  in 
the  bottom  center  of  the  screen). 
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Fig.  5  The  WMI  shows  the  OM  overlay  on  a  forward-facing  3D  map.  This  is  a  ground-level  view  of 
the  environment;  red  and  green  squares  represent  edges  of  obstacles.  Red  indicates  obstacles 
that  cannot  be  passed  through  (e.g.,  a  wall);  green  represents  obstacles  that  can  potentially  be 
traversed  by  the  UGV  (e.g.,  a  tunnel). 


Fig.  6  The  WMI  shows  the  OM  overlay  on  a  top-down,  2-dimensional  (2D)  map  in  an  aerial  view 
of  the  environment.  The  edges  of  obstacles  are  represented  with  red  pixels;  areas  shaded 
gray  are  “unknown”  because  the  view  from  the  UGV  is  obstructed  by  the  elements  in  the 
environment  (e.g.,  a  wall). 
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Fig.  7  Image  from  live  video  during  the  SOURCE  ATO  event:  WMI  shows  TP  (green  and  blue 
line)  and  OM  (red  and  green  pixels)  in  overhead  (2D)  view.  The  OM  helps  outline 
buildings/structures  that  might  otherwise  be  obscured. 

The  WMI  display  has  2  screens  showing  different  views  of  the  environment.  The  forward-facing 
video  display  is  referred  to  as  the  3D  driving  display.  This  display  shows  a  live,  forward-facing 
feed  of  the  environment  directly  in  front  of  the  UGV  and  allows  for  a  ground-level  “3D”  view  of 
the  mission  space.  An  overhead  map  view  of  the  environment,  referred  to  as  the  2D  map  display, 
shows  an  overhead  “satellite”  view  of  the  environment.  The  2D  view  also  gives  users  an 
indication  of  the  UGV’s  location  within  the  mission  space.  Examples  of  the  2D  portion  of  the 
WMI  display  can  be  seen  in  Figs.  6  and  7.  The  2D  and  3D  screens  were  continuously  displayed 
on  two  17 -inch  monitors  simultaneously  so  that  participants  could  attend  to  either  screen  at  any 
time  during  the  experiment.  Participants  primarily  monitored  the  forward-facing  camera  of  the 
simulated  UGV  and  entered  task  information  using  a  touch-screen  monitor.  All  of  the  software 
was  run  on  standard  personal  computers,  using  two  standard  17-inch  color  monitors  with  touch¬ 
screen  capability. 

Autonomous  navigation  for  the  simulated  UGV  was  handled  using  the  Autonomous  Navigation 
System  (ANS)  software.  This  software  was  identical  to  the  software  used  for  a  real-world 
version  of  the  autonomous  UGV  and  allowed  the  vehicle  to  navigate  an  environment  via 
waypoint  plan  while  avoiding  environmental  obstacles  (GDRS  2013). 
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2.2  Participants 

For  this  study,  enlisted  personnel,  including  noncommissioned  officers,  from  the  594th 
Transportation  Company  were  recruited  as  participants.  A  total  of  8  individuals  participated. 
Participants  were  all  male  with  an  average  age  of  27  years  (standard  deviation  =  4.73).  The  8 
participants  were  all  active-duty  Soldiers  with  experience  in  infantry  and  motor-pool  missions. 

2.3  Materials  and  Tests 

The  experiment’s  participants  were  required  to  complete  assessments  of  their  color- vision  acuity, 
spatial-visualization  ability,  and  spatial-orientation  ability  as  well  as  a  demographic 
questionnaire  that  was  used  to  characterize  the  participant  population. 

To  assess  color-vision  acuity,  the  Ishihara  (1917)  color-vision  test  was  used  (see  Appendix  A)  as 
a  screening  procedure.  The  color -vision  test  was  used  to  ensure  that  participants  have  no 
unknown  color-vision  deficiencies,  as  color  is  an  important  component  of  the  WMI  and  operator 
aids. 

Two  aspects  of  the  Soldiers’  spatial  ability — spatial  visualization  and  spatial  orientation — were 
assessed  using  the  relevant  sections  of  the  Guilford-Zimmerman  (Guilford  1956)  attribute 
survey  (see  Appendix  B  and  Appendix  C).  Both  of  these  assessments  have  been  used  previously 
in  studies  of  human-robot  interaction  (HRI)  (e.g.,  Fincannon  et  al.  2008),  and  have  shown 
positive  corollary  effects  with  robot-operator  performance.  These  results  indicated  that 
individuals  displaying  higher  levels  of  spatial  ability  generally  show  higher  levels  of 
performance  when  engaging  in  FIRI  supervision  tasks.  The  spatial-ability  scores  were  used  as 
covariates  (as  needed)  in  the  data  analysis  for  this  study. 

Participants’  perceived  workloads  were  evaluated  after  the  completion  of  each  mission  scenario 
using  the  NASA-TLX  questionnaire  (Hart  and  Staveland,  1988;  Appendix  D).  The  NASA  TLX 
is  a  self-report  questionnaire  of  perceived  demands  in  6  areas:  mental,  physical,  temporal,  effort 
(mental  and  physical),  frustration,  and  performance.  Participants  were  asked  to  rate  their 
perceived  workload  level  in  the  6  areas  on  100-point  scales.  The  ratings  were  used  to  quantify 
the  perceived  changes  in  workload  for  the  various  operator  aids  used. 

Finally,  participants’  operator-aid  configuration  preference  was  recorded  after  each  trial  by 
asking  each  participant  to  identify  their  favorite  and  least  favorite  aid  configurations . 

2.4  Experimental  Design 

Participants  in  this  study  completed  a  series  of  reconnaissance  missions  in  which  they  detected 
targets  from  a  simulated  autonomous  UGV  platform  traversing  within  the  simulated  MOUT 
environment.  Mission  navigation  of  the  UGV  was  fully  autonomous;  however,  the  UGV 
platform  interface  included  a  variety  of  operator  aids  intended  to  inform  the  operator  about  robot 
intent  and  robot  obstacle  identification.  Participants  took  teleop  (remote)  control  of  the  UGV 
only  when  it  was  encountering  an  obstacle  (discussed  further  in  Section  2.5  of  this  report). 
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This  study  employed  a  2  (Travel  Planner  Condition)  x  2  (Obstacle  Map  Condition)  x  2  (Reroute 
Alert  Condition)  within  subjects  design  (see  Table  1).  For  analytical  purposes,  each  experimental 
cell  was  compared  with  the  control  cell  using  paired  sample  /-tests.  This  analysis  allowed 
researchers  to  understand  if  operator  aids  presented  as  a  single  aid  or  multiple  aids  contributed  to 
improved  performance  over  the  condition  involving  no  aids. 

Table  1  Experimental  Design  Matrix.  The  design  is  a  2  x  2  x  2  within-subjects  design,  with  Travel  Planner, 

Reroute  Alerts,  and  Obstacle  Map  overlays  as  the  3  independent  variables.  All  8  participants  experienced 
all  8  conditions  twice  over  2  sessions  spanning  2  days. 


With  Travel  Planner 

Without  Travel  Planner 

With 

Reroute  Alerts 

Without 
Reroute  Alerts 

With 

Reroute  Alerts 

Without 
Reroute  Alerts 

With 

Obstacle  Map 
Overlays 

Without  Obstacle 
Map  Overlays 

2.4.1  Independent  Variables 

Travel  Planner  (TP),  Obstacle  Map  (OM),  and  Rerouting  Alert  (RA)  represent  the  3  independent 
variables.  Each  of  the  independent-variable  levels  for  the  3  Operator  Aid  conditions  includes  a 
“with”  and  “without”  condition. 

2.4. 1.1  Travel  Planner 

The  TP  was  adapted  from  a  combination  of  the  Short  Term  Planner  and  Long  Term  Planner  used 
in  an  early  laboratory  study  looking  at  operator  aids  (Evans  2013).  In  the  previous  study,  the 
Short  Term  Planner  visually  showed  the  planned  robot  path  for  the  next  3  to  5  seconds  (s)  and 
was  shown  as  a  green  line.  The  Long  Term  Planner  showed  the  robot  path  planned  further  out  in 
time,  such  as  in  the  next  minutes,  as  a  blue  line.  (See  Fig.  8  for  an  example  of  each.)  The  TP  for 
this  experiment  was  adapted  from  the  Long  and  Short  Term  Planners  in  response  to  users’ 
recommendations  of  finding  both  the  Long  and  Short  Term  Planners  useful,  but  with  concerns 
expressed  about  the  overall  amount  of  clutter  on  the  screen  obscuring  view  of  the  mission 
environment.  The  TP  was  represented  by  a  green-  and  blue-colored  translucent  line  overlaid  on 
both  forward-facing  3D  driving  display  and  2D  overhead-map  display  of  the  WMI.  The 
translucent  line  extended  out  from  the  vehicle’s  position,  providing  information  about  where  the 
vehicle  intended  to  travel.  The  TP  was  generated  using  data  from  the  ANS  about  the  path  which 
the  automation  is  attempting  to  follow.  The  green  portion  of  the  line  represented  the  path  planned 
for  the  next  3-5  s,  while  the  blue  portion  of  the  single  translucent  line  conveyed  the  path  planned 
further  out  in  time.  The  actual  length  of  the  line  was  dependent  upon  vehicle  speed  and  varied 
accordingly.  See  Figs.  3,  4,  and  7  for  examples  of  the  TP  display. 
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Fig.  8  Example  of  the  original  WMI  3D  interface  showing  both  the  Short  Term  Planner 
(green  line)  and  Long  Term  Planner  (blue  line)  operator  aids  (Evans  2013).  hi 
response  to  user  feedback  that  identified  screen  clutter  as  a  concern,  these  2  separate 
lines  were  combined  into  the  single  TP  operator  aid. 

2.4. 1.2  Obstacle  Map 

Obstacle  maps  were  generated  from  Laser  Radar  data  acquired  by  the  ANS,  which  were  then 
transformed  to  display  the  outlines  of  physical  obstacles  in  the  environment.  Scenarios  with  the 
Obstacle  Map  included  translucent  overlays  on  both  forward-facing  (3D)  video  stream  and 
overhead  map  (2D)  views  showing  obstacles  that  the  UGV  has  detected.  On  the  3D  forward¬ 
facing  view,  the  OM  was  represented  by  red  (impassable  obstacle)  and  green  (passable  obstacle) 
squares  at  ground  level  (see  Figs.  3  and  5).  For  the  2D  overhead  view,  OMs  were  represented 
using  red  pixels  to  show  nonpassable  obstacle  borders  and  grey  translucent  shading  to  indicate 
unknown  areas  of  the  map  (see  Figs.  6  and  7).  Overhead  maps  were  pre-set  at  a  fixed  zoom  scale 
to  ensure  that  all  participants  received  the  same  view.  The  overhead  2D-map  view  was  not 
adjustable  by  the  operator. 

2.4. 1.3  Rerouting  Alert 

The  Rerouting  Alert  works  much  like  the  RA  on  commercially  available  GPS  navigational 
systems.  The  autonomous  UGV  has  a  planned  route  upon  which  it  intends  to  travel  to  get  from 
point  A  to  point  B.  Whenever  the  UGV  diverges  from  that  intended  route  (e.g.,  it  makes  a  left 
turn  rather  than  a  right  turn),  a  visual  alert  message  will  be  produced  that  briefly  displays  on  the 
forward-facing  view  screen  of  the  WMI  (see  Fig.  9).  The  alert  message  is  then  placed  into  an 
alert-messaging  cue  which  can  be  referenced  at  any  time  after  the  alert.  During  the  conditions 
when  an  RA  is  present,  an  alert  is  only  displayed  visually,  without  audio  cues. 
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Fig.  9  Example  of  the  Rerouting  Alert  in  use  with  the  forward-facing  (3D)  view  of  the 
WMI:  The  UGV  is  changing  its  original  route  to  maneuver  around  a  perceived 
obstacle.  RA  shows  briefly  in  a  conspicuous  location  (center  screen)  in  red  to 
ensure  saliency  of  the  alert.  Image  was  taken  directly  from  live  video  recorded 
during  the  SOURCE  ATO  event. 

2.4.2  Dependent  Variables 

Several  types  of  dependent  measures  were  seen  as  important  for  this  study.  Measures  related  to 
both  performance  and  workload  were  used.  The  reconnaissance-mission  scenarios  had  the 
primary  performance  measure  of  target  detection.  Measures  related  to  teleoperation  were  also 
used.  Although  the  UGVs  are  autonomous,  there  were  times  when  operator  intervention  was 
necessary.  The  purpose  of  the  operator  aids  was  to  provide  current  and  future  robot  status  and 
provide  insights  to  the  operator  about  the  reasons  for  various  robot  actions.  If  the  operator  aids 
are  helping  the  operator  to  better  understand  and  anticipate  UGV  actions,  it  is  thought  that 
operator  interventions  (i.e.,  manual  teleoperation)  will  be  less  frequent  and  will  take  less  time 
than  without  the  information  provided  by  the  operator  aids.  The  less  time  the  operator  spends 
supervising  the  UGV  through  teleoperation,  the  more  time  there  will  be  to  conduct  the  primary 
task  of  IED  detection. 

Dependent  variables  for  this  study  included  1)  the  number  of  accurate  target  detections,  2)  the 
number  of  times  engaging  teleoperation  mode,  3)  the  total  amount  of  time  spent  in  teleoperation 
mode,  and  4)  the  total  mission  time.  These  are  all  variants  of  well-established  metrics  of 
performance  used  in  HRI  research  (Fong  et  al.  2004).  “Number  of  accurate  target  detections”  is 
defined  as  the  number  of  correct  target  locations  selected  on  the  overhead  map  during  the 
performance  mission.  “Number  of  times  engaging  teleoperation  mode”  is  defined  as  the  total 
number  of  times  the  participant  engages  the  teleoperation  mode  of  UGV  control  during  a 
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performance  mission.  (Teleoperation  mode  is  engaged  by  a  button  control  on  the  right  side  of  the 
WMI  display.)  “Total  time  spent  in  teleoperation  mode”  is  defined  as  the  total  amount  of  time  in 
seconds  that  operators  spent  in  teleoperation  mode  as  opposed  to  autonomous  mode.  (Since 
teleoperation  is  engaged  and  disengaged  with  a  button,  an  accurate  time  can  be  recorded.)  “Total 
mission  time”  is  defined  as  the  total  amount  of  time  in  seconds  it  takes  to  complete  a  mission 
beginning  at  the  moment  an  automated  route  plan  has  begun  executing  until  the  completion  of 
that  plan,  including  all  pauses  and  occurrences  of  teleoperation  control.  All  of  this  information 
was  gathered  as  .log  files  created  for  each  mission.  Time  and  frequency  of  occurrence  data  were 
obtained  from  the  .log  files  and  input  into  statistical  software  for  further  analysis. 

The  NASA  TLX  was  used  to  measure  ratings  of  perceived  workload,  resulting  in  an  unweighted 
composite  score  derived  from  6  areas  of  workload:  mental,  physical,  temporal,  effort,  frustration, 
and  performance.  In  addition,  data  were  collected  about  participants’  spatial  abilities  (via  the 
Guilford-Zimmerman  Spatial  Aptitude  survey)  to  be  used  as  covariates  in  data  analysis.  The 
spatial-ability  covariate  score  was  created  by  adding  the  scores  from  the  2  separate  spatial-ability 
surveys  (described  in  Section  2.3)  together  into  a  single  composite  score. 

Finally,  participants’  preferences  for  favorite  and  least  favorite  operator  aid  were  recorded.  These 
data  were  gathered  by  simply  asking  each  participant  at  the  end  of  the  experiment  “Which 
condition  was  your  favorite?”  and  “Which  condition  was  your  least  favorite?” 

2.5  Procedure 

Once  an  informed  consent  was  completed,  each  Soldier  participant  was  given  the  color-vision 
test.  Any  participant  who  exhibited  any  color  vision  impairment  was  removed  from  the  study. 
Following  this,  each  participant  was  given  a  demographic  questionnaire  to  fill  out.  Finally,  the 
participant  was  given  the  2  assessments  of  spatial  ability:  one  for  spatial  visualization  and  one 
for  spatial  orientation. 

Each  participant  then  was  given  a  short  tutorial  on  the  use  of  the  simulation  interface  and 
instructed  on  how  to  complete  the  reconnaissance  task.  The  simulation  was  a  representation  of  a 
UGV  traversing  a  predetermined  route  within  the  MOUT  simulated  environment.  The  participant 
monitored  the  displays,  searching  for  targets  (represented  as  simulated  IEDs — barrels  with  timers 
on  them — that  were  not  obscured  from  view)  as  the  UGV  traveled  along  the  route. 

The  participant  was  given  an  opportunity  to  observe  the  2D  and  3D  displays  simultaneously  and 
to  use  the  simulation  system  in  a  practice  scenario  to  become  familiar  with  the  system.  This  gave 
each  participant  an  opportunity  to  review  the  various  target  and  obstacle  items  that  would  be 
encountered  during  the  experimental  scenarios.  The  training  session  lasted  about  2.5  h  on  the 
first  day. 

After  familiarization  and  training,  each  participant  was  asked  to  act  as  operator  for  a  UGV  on  a 
reconnaissance  mission.  Each  participant  completed  8  counterbalanced  experimental  scenarios 
twice  for  a  total  of  16  trials,  exploring  all  of  the  possible  independent  variable  combinations  in 
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the  2  (TP  Condition)  x  2  (OM  Condition)  x  2  (RA  condition)  within  subjects  design,  over  the 
course  of  2  experimental  sessions  of  about  2  h  each  on  consecutive  days. 

Each  reconnaissance  scenario  consisted  of  one  UGV  following  a  predetermined  route  through 
the  urban  terrain.  Along  the  route,  the  participant  had  the  opportunity  to  detect  up  to  6  separate 
targets  (i.e.,  simulated  IEDs).  In  addition  to  scanning  for  targets,  each  participant  monitored  the 
UGV  to  ensure  it  remained  on  its  intended  path.  Each  mission  included  a  set  of  obstacles  (3  total) 
that  caused  the  UGV’s  ANS  to  deviate  from  its  planned  path.  These  obstacles — patches  of  tall 
grass — were  positioned  randomly  throughout  the  route.  This  obstacle  type  had  been  shown 
previously  (Evans  2013)  to  cause  the  most  inconsistent  ANS  behavior  because  it  was  the  most 
difficult  obstacle  for  the  ANS  to  correctly  interpret.  In  addition  to  the  patches  of  grass  that  would 
cause  a  route  deviation,  distracter  obstacles  (also  patches  of  grass)  were  included  that  would  not 
cause  any  route  deviations.  Within  the  scenarios,  the  patches  of  tall  grass  represented  ambiguous 
situations  where  the  autonomy  might  incorrectly  interpret  the  UGV’s  ability  to  proceed.  In  these 
cases,  the  operator  aids  might  be  most  useful  by  helping  the  operator  see  and  understand  what 
the  robot  plans  were  with  respect  to  obstacles  and  travel  paths. 

Participants  were  asked  to  choose  an  appropriate  course  of  action  (COA)  whenever  an  obstacle 
was  encountered.  Two  possible  COAs  were  available:  1)  let  the  UGV  continue  on  its  own  with 
no  operator  intervention,  or  2)  the  operator  intervenes  by  using  the  teleoperation  control  to  move 
the  UGV  around  the  obstacle  and,  once  it  is  cleared,  continue  on  the  planned  travel  path  by  re¬ 
engaging  the  UGV’s  autonomous-navigation  mode.  When  tall  grass  was  encountered  and  caused 
a  route  deviation  from  the  plan,  the  appropriate  COA  was  to  instruct  the  UGV  to  continue  on  the 
planned  route  without  stopping,  with  no  operator  intervention  (COA  #1).  COAs  were  executed 
by  speaking  aloud  that  an  obstacle  had  been  encountered  and  then  operating  the  UGV  with  the 
appropriate  maneuver  for  the  obstacle,  either  “continue”  or  “teleoperation”.  At  the  end  of  each 
of  the  16  scenarios,  TLX  data  were  collected  to  obtain  a  measure  of  perceived  workload  for  that 
scenario.  At  the  completion  of  all  of  the  scenarios,  user-preference  data  were  collected  via 
interview. 


3.  Performance  Data  Results  and  Analysis 


To  investigate  the  performance -related  hypotheses  (Hi  and  H2),  an  analysis  of  variance 
(ANOVA)  was  completed  for  each  of  the  dependent  variables.  Within-subjects  ANOVAs  were 
conducted  using  SPSS  19.0  software  (IBM  2013).  Some  of  the  data  files  were  corrupted  and 
these  data  points  were  eliminated  via  pairwise  deletion.  Results  of  the  individual  ANOVAs  are 
presented  in  the  following  sections. 
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3.1  Target  Detection 


A  within-subjects  ANOVA  was  conducted  to  evaluate  the  effect  of  operator -aid  condition  on 
participants’  ability  to  accurately  detect  targets  in  the  experimental  environment.  Results  of  the 
ANOVA  showed  no  significant  differences  in  the  experimental  conditions  ( F(7,98 )  =  1.986, 
p  =  .071).  To  ensure  that  the  individual  difference  factors  related  to  spatial  abilities  were  not 
influencing  the  analysis,  covariate  data  were  used  in  a  follow-up  analysis  of  covariance 
(ANCOVA).  Covariates  were  derived  from  the  2  spatial  ability  evaluations:  spatial  orientation 
and  spatial  visualization.  The  co variates,  representing  an  individual  participant’s  spatial  ability, 
were  derived  from  the  total  number  of  correct  answers  for  both  spatial  assessments,  resulting  in  a 
composite  spatial-ability  score.  The  covariate  analysis  was  used  to  control  for  spatial  ability, 
since  it  is  known  that  spatial  ability  is  positively  correlated  with  robotic -operator  performance 
(Fincannon  et  al.  2008).  Therefore,  the  results’  failure  to  show  differences  in  performance  should 
be  attributed  to  the  experimental  conditions  (i.e.,  operator  aids)  and  not  to  any  differences  in 
participants’  innate  abilities.  The  same  approach  for  covariates  was  used  for  each  of  the 
performance-measure  analysis  (Sections  3. 1-3.4).  Results  of  the  ANCOVA  showed  no 
significant  influence  from  any  of  the  covariates  and  no  significant  differences  in  the  results 
( F(7,97 )  =  0.537,  p  =  .821).  The  data  from  this  analysis  have  been  summarized  in  Table  2. 

Table  2  Means  and  standard  deviations  for  Target  Detection  data  in  the  8  conditions  (measured  in  correct 

detections  per  mission;  i.e.,  larger  score  is  better;  maximum  is  6).  No  significant  differences  were  found 
between  conditions. 


Dependent  Variable:  Target  Detection 

Condition 

Mean 

Standard  Deviation 

C 1 :  Travel  Planner,  Reroute  Alert,  &  Obstacle  Map 

5.15 

0.80 

C2:  Travel  Planner  &  Obstacle  Map 

4.85 

1.07 

C3:  Travel  Planner  &  Reroute  Alert 

5.31 

0.63 

C4:  Travel  Planner  only 

5.62 

0.51 

C5:  Reroute  Alert  &  Obstacle  Map 

4.85 

1.41 

C6:  Obstacle  Map  only 

5.08 

0.86 

C7 :  Reroute  Alert  only 

5.62 

0.51 

C8:  No  Operator  Aids 

5.38 

0.87 

3.2  Teleoperation  Occurrences 

A  within-subjects  ANOVA  was  conducted  to  evaluate  the  effect  of  operator -aid  condition  on 
participants’  use  of  the  teleoperation  control  mode  in  the  experimental  environment.  Results  of 
the  ANOVA  showed  that  significant  main  effect  among  the  experimental  conditions  did  exist 
( F(7,98 )  =  3.261,  p  =  .004,  rf  =  .207).  The  data  from  this  analysis  have  been  summarized  in 
Table  3. 
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Table  3  Means  and  standard  deviations  for  Teleoperation  Occurrences  in  the  8  conditions  are  measured  in 
number  of  times  teleoperation  mode  was  engaged  per  mission  (i.e.,  smaller  score  is  better,  no 
maximum).  Significant  differences  were  found  between  conditions. 


Dependent  Variable:  Teleoperation  Occurrences 

Condition 

Mean 

Standard  Deviation 

C 1 :  Travel  Planner,  Reroute  Alert,  &  Obstacle  Map 

5.15 

1.52 

C2:  Travel  Planner  &  Obstacle  Map 

4.46 

0.97 

C3:  Travel  Planner  &  Reroute  Alert 

4.38 

2.14 

C4:  Travel  Planner  only 

4.23 

1.09 

C5:  Reroute  Alert  &  Obstacle  Map 

4.85 

1.63 

C6:  Obstacle  Map  only 

6.23 

3.22 

C7 :  Reroute  Alert  only 

5.15 

1.41 

C8:  No  Operator  Aids 

6.85 

1.46 

Post  hoc  analysis,  using  pair-wise  comparison  with  a  Bonferroni  correction,  showed  that  3 
experimental  conditions  produced  significantly  lower  numbers  of  teleoperation  occurrences  than 
the  control  condition:  Condition  2  (Travel  Planner  and  Obstacle  Map),  Condition  3  (Travel 
Planner  and  Reroute  Alerts),  and  Condition  4  (Travel  Planner  only)  were  all  significantly  lower 
than  the  control  condition  (no  aids)  at  the  .05  level.  No  other  results  showed  significance.  These 
results  are  shown  in  Fig.  10,  with  the  3  conditions  significantly  lower  (identified  by  the  green 
arrows)  than  the  control  condition  (identified  by  the  red  arrow). 


Fig.  10  Means  plot  for  Teleoperation  Occurrences  by  experimental 

condition:  fewer  teleoperation  occurrences  is  interpreted  as  better 
performance.  Note:  “All”  =  Travel  Planner  (TP),  Obstacle  Map 
(OM),  and  Rerouting  Alert  (here,  RR);  “None”  =  Control — no 
operator  aids. 
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3.3  Teleoperation  Time 


A  within-subjects  ANOVA  was  conducted  to  evaluate  the  effect  of  the  operator-aid  condition  on 
participants’  teleoperation-control  times  throughout  the  UGV’s  route  in  the  experimental 
environment.  Results  of  the  ANOVA  showed  significant  main  effect  did  exist  among  the 
experimental  conditions  ( F(7,98 )  =  3.546,  p  =  .003,  if  =  .279).  The  data  from  this  analysis  have 
been  summarized  in  Table  4. 

Table  4  Means  and  standard  deviations  for  Teleoperation  Time  in  the  8  conditions  are  measured  in  number  of 
seconds  per  mission  (i.e.,  smaller  score  is  better).  Significant  differences  were  found  between 
conditions. 


Dependent  Variable:  Teleoperation  Time 

Condition 

Mean 

Standard  Deviation 

C 1 :  Travel  Planner,  Reroute  Alert,  &  Obstacle  Map 

65.50 

19.16 

C2:  Travel  Planner  &  Obstacle  Map 

59.80 

22.62 

C3:  Travel  Planner  &  Reroute  Alert 

50.60 

17.94 

C4:  Travel  Planner  only 

55.60 

18.51 

C5:  Reroute  Alert  &  Obstacle  Map 

59.20 

19.23 

C6:  Obstacle  Map  only 

74.00 

35.65 

C7 :  Reroute  Alert  only 

67.50 

23.45 

C8:  No  Operator  Aids 

111.20 

70.41 

Post  hoc  analysis  showed  the  experimental  conditions  produced  an  average  decrease  of  49.5  s  in 
amount  of  time  spent  in  teleoperation  mode  when  compared  to  the  control  (no  aid)  condition. 
Using  paired  sample  t- tests  and  Tukey  HSD  corrections  for  Type  I  error,  results  showed  that  5  of 
the  experimental  conditions  (Conditions  1-5)  had  significantly  lower  time  spent  in  teleoperation 
mode  than  the  control: 

•  Condition  1 — Travel  Planner,  Obstacle  Map,  and  Reroute  Alerts,  f(13)  =  1 .810,  p  =  .048, 
one  tailed; 

•  Condition  2 — Travel  Planner  and  Obstacle  Map,  t(14)  =  2.071,  p  =  .028,  one  tailed; 

•  Condition  3 — Travel  Planner  and  Reroute  Alerts,  f(13)  =  2.186,  p  =  .024,  one  tailed; 

•  Condition  4 — Travel  Planner  only,  f(13)  =  2.478,  p  =  .014,  one  tailed;  and, 

•  Condition  5 — Obstacle  Map  and  Reroute  Alerts,  f(15)  =  2.377,  p  =  .016,  one  tailed. 

No  other  results  showed  significance.  These  results  are  shown  in  Fig.  1 1 ,  with  the  mean  for  the 
experimental  conditions  lower  (identified  by  the  green  line)  than  the  control  condition  (identified 
by  the  red  line)  by  an  average  total  of  49.5  s. 
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Fig.  1 1  Means  plot  for  Total  Teleoperation  Time  by  experimental  condition: 

green  line  represents  the  average  time  for  all  operator-aid  conditions; 
red  line  shows  the  average  time  for  the  control  condition.  Difference 
between  the  red  and  green  lines  =  49.5  s.  Note:  “All”  =  Travel 
Planner  (TP),  Obstacle  Map  (OM),  and  Rerouting  Alert  (here,  RR); 
“None”  =  Control — no  operator  aids. 


3.4  Total  Mission  Time 

Finally,  a  within- subjects  ANOVA  was  conducted  to  evaluate  the  effect  of  operator-aid 
condition  on  overall  mission  time.  Results  of  the  ANOVA  showed  no  significant  differences  in 
the  experimental  conditions  ( F(7,113 )  =  0.937,  p  =  .483).  To  ensure  that  other  factors  were  not 
influencing  the  analysis,  covariate  data  were  used  in  a  follow-up  ANCOVA  analysis.  Covariates 
were  derived  from  the  spatial -ability  evaluations,  and  the  analysis  process  was  the  same  as 
described  above.  Results  of  the  ANCOVA  showed  no  significant  influence  from  any  of  the 
covariates  and  no  significant  differences  in  the  results  ( F(7,112 )  =  0.697,  p  =  .677).  The  data 
from  this  analysis  have  been  summarized  in  Table  5. 
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Table  5  Means  and  standard  deviations  for  Total  Mission  Time  data  in  the  8  conditions  are  measured  in  total 
seconds  per  mission,  from  beginning  of  automated  plan’s  execution  to  completion  of  plan  (i.e.,  less 
time  is  better).  No  significant  differences  were  found  between  conditions. 


Dependent  Variable:  Total  Mission  Time 

Condition 

Mean 

Standard  Deviation 

C 1 :  Travel  Planner,  Reroute  Alert,  &  Obstacle  Map 

514.69 

125.69 

C2:  Travel  Planner  &  Obstacle  Map 

525.08 

137.45 

C3:  Travel  Planner  &  Reroute  Alert 

518.85 

117.58 

C4:  Travel  Planner  only 

525.31 

88.92 

C5:  Reroute  Alert  &  Obstacle  Map 

508.08 

68.40 

C6:  Obstacle  Map  only 

574.69 

164.55 

C7 :  Reroute  Alert  only 

544.85 

95.70 

C8:  No  Operator  Aids 

593.38 

125.33 

4.  Workload  Data  Analysis 


To  investigate  the  workload  hypothesis  (H3),  comparisons  of  the  experimental  conditions  were 
conducted  using  analysis  of  variance.  The  NASA-TLX  composite  scores  were  calculated  by 
summing  the  6  subscale  scores  and  dividing  by  6  (as  suggested  by  Hill  et  al.  1992;  highest 
possible  score  is  100).  The  composite  scores  were  analyzed  with  a  one-way  ANOVA  using  SPSS 
19.0.  The  composite  scores  reflected  the  self-reported  workload  score  for  each  operator-aid 
condition.  Results  of  the  ANOVA  showed  no  significant  differences  existed  ( F(7 ,113 )  =  1.195, 
p  =  .31 1)  among  the  self-reported  workload  measures  for  the  different  operator -aid  conditions. 
These  results  are  in  Table  6  and  Fig.  12. 

Table  6  Means  and  standard  deviations  for  the  composite  NASA-TLX  subjective  workload  are  measured  by 
experimental  condition:  maximum  =  100,  with  lower  ratings  reflecting  less  workload.  No  significant 
differences  were  found. 


Dependent  Variable:  Subjective  Workload 

Condition 

Mean 

Standard  Deviation 

C 1 :  Travel  Planner,  Reroute  Alert,  &  Obstacle  Map 

38.32 

12.17 

C2:  Travel  Planner  &  Obstacle  Map 

38.81 

11.37 

C3:  Travel  Planner  &  Reroute  Alert 

38.53 

10.06 

C4:  Travel  Planner  only 

40.30 

8.97 

C5:  Reroute  Alert  &  Obstacle  Map 

42.47 

12.40 

C6:  Obstacle  Map  only 

41.32 

12.81 

C7 :  Reroute  Alert  only 

41.68 

14.02 

C8:  No  Operator  Aids 

39.41 

11.28 
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Fig.  12  Means  plot  for  NAS  A- TLX  workload  composite  scores  by 

experimental  condition:  range  of  possible  scores  is  0-100.  There  are 
no  significant  differences  among  conditions.  Note:  “All”  =  Travel 
Planner  (TP),  Obstacle  Map  (OM),  and  Rerouting  Alert  (here,  RR); 
“None”  =  Control — no  operator  aids. 


5.  User-Preference  Data  Analysis 


5.1  Operator  Aid  Condition 

Participants  were  asked  which  of  the  conditions  were  their  favorite  and  least  favorite  to  use  (the 
post-study  interview  questions  described  in  Section  2.4.2).  While  the  choices  for  favorite 
condition  were  more  varied,  the  majority  of  the  choices  (by  5  of  the  8  participants)  were  for 
experimental  conditions  involving  the  Travel  Planner  operator  aid.  Additionally,  half  (4  of  8) 
listed  the  control  condition  with  “no  operator  aids”  as  their  least  favorite.  (See  Table  7.) 
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Table  7  Frequency  of  responses  for  participants’  favorite  and  least  favorite  operator-aid  configurations  (n  =  8). 


Dependent  Variable:  User  Preferences 

Conditions: 

Favorite  Condition 
No.  of  Responses  n=8 

Least  Favorite 
Condition 

No.  of  Responses  n=8 

C 1 :  Travel  Planner,  Reroute  Alert,  &  Obstacle  Map 

1 

0 

C2:  Travel  Planner  &  Obstacle  Map 

1 

0 

C3:  Travel  Planner  &  Reroute  Alert 

1 

0 

C4:  Travel  Planner  only 

2 

2 

C5:  Reroute  Alert  &  Obstacle  Map 

0 

0 

C6:  Obstacle  Map  only 

3 

0 

C7 :  Reroute  Alert  only 

0 

2 

C8:  No  Operator  Aids 

0 

4 

When  asked  about  their  choices  and  what  influenced  them,  the  Soldiers’  most  common  concern 
was  clutter  on  the  interface.  The  Soldiers  involved  in  this  study  were  very  clear  that  too  many 
operator  aids  could  obscure  their  view  of  the  environment.  The  Soldiers  emphasized  they 
extracted  as  much  information  as  possible  from  a  single  aid.  This  is  represented  in  the  choice  of  a 
single-aid  condition  as  the  favorite  for  5  of  the  8  participants. 


6.  Discussion 


6.1  Performance 

Results  of  the  performance-data  analysis  support  the  hypothesis  related  to  teleoperation-mode 
use  (H2).  Participants  used  teleoperation  controls  less  frequently  and  for  less  time  in  the 
experimental  conditions  than  in  the  control  condition.  Therefore,  it  appears  that  the  participants 
were  able  to  choose  the  most  appropriate  COA  and  allow  the  autonomy  to  continue  to  operate 
when  appropriate  in  the  experimental  conditions;  the  participant  operators  did  not  preemptively 
use  teleoperation  control.  This  suggests  that  operators  were  able  to  more  accurately  predict  the 
behavior  of  robotic  assets  when  provided  with  additional  robotic  intent  information.  The 
scenarios  for  this  study  were  relatively  short  (about  10  min).  We  speculate  that  for  longer 
missions,  this  finding — reduction  of  time  in  direct  control  of  a  robotic  asset  means  more  time 
(and  potentially  more  cognitive  resources)  for  the  operator  to  focus  on  the  primary  target- 
detection  task — could  be  even  more  significant. 

However,  the  hypothesis  that  the  primary  task  of  detecting  targets  would  improve  with  the  use  of 
operator  aids  (Hi)  was  not  supported.  This  failure  to  observe  more  target  detections  with  the  use 
of  operator  aids  may  be  due  to  several  factors.  A  relatively  small  sample  size  (n  =  8)  and  low 
workload  task  could  have  affected  results.  The  task  of  supervising  a  single  robot  without 
additional  tasks  allowed  the  participants  to  focus  a  large  amount  of  their  attention  on  the  primary 
task  (target  detection).  Including  a  demanding  secondary  task  (such  as  radio  communication)  or 
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an  increase  in  the  number  of  assets  (i.e.,  robots  or  other  tactical  assets)  being  supervised  would 
be  expected  to  cause  higher  workload.  Operator  aids  may  have  a  greater  positive  performance 
effect  in  higher-workload  conditions.  Additionally,  participants  in  this  study  received  limited 
training  and  were  exposed  to  the  operator  aids  for  a  relatively  short  amount  of  time  during  the 
data-collection  scenarios.  More  training  and  exposure  might  help  users  understand  more 
“strategic”  uses  for  the  operator  aids — such  as,  using  the  Obstacle  Map  to  help  identify  potential 
target  locations — rather  than  just  to  understand  what  the  robot  sensors  are  observing.  Developing 
a  strategy  for  the  use  of  operator  aids  could  potentially  improve  overall  performance. 

6.2  Workload 

Results  of  the  workload-data  analysis  did  not  support  the  hypothesis  (H3)  associated  with  the 
perceived  workload  measurements.  Participants  did  not  show  any  significant  difference  in 
perceived  workload  between  the  experimental  and  control  conditions.  This  failure  to  observe  any 
significant  differences  may  be  due  to  several  factors.  The  relatively  small  sample  size  (n  =  8)  and 
simple  task  (i.e.,  looking  for  IEDs  that  were  not  obscured  from  view)  could  have  affected  results. 
Additionally,  we  observed  that  the  Soldiers  who  participated  in  this  study  may  have  experienced 
some  testing  fatigue,  which  perhaps  led  to  a  smaller  range  of  (i.e.,  less  contrasting)  perceptions 
on  the  NASA-TLX  workload  survey.  The  workload-survey  results  suggest  that  some  participants 
began  repeating  survey  answers  for  each  scenario  rather  than  providing  more  thoughtful  and 
condition-specific  perceptions  of  workload.  It  should  be  noted  that  while  the  use  of  operator  aids 
did  not  reduce  perceived  workload,  neither  did  the  inclusion  of  aids  increase  the  participants’ 
perceived  workload. 

Further,  the  overall  workload  results  remained  relatively  low  across  conditions  throughout  the 
study.  This  suggests  that  the  task  itself  may  have  been  not  challenging  enough  to  produce 
significant  workload  challenges  for  the  participants.  If  the  task  was  too  easy,  it  may  explain  why 
the  covariate  measures,  which  have  been  shown  to  be  significant  in  previous  studies,  did  not 
yield  any  significant  results  here.  That  is,  all  participants  could  perform  the  tasks  well,  regardless 
of  their  individual  spatial  ability  (which  in  this  case  could  be  considered  average  overall).  In 
future  studies,  task  difficulty  should  be  advanced  to  levels  that  ensure  an  appropriate  amount  of 
workload  so  that  “floor  effects”,  such  as  those  noted  here,  are  not  a  factor.  This  would  likely  be 
achieved  by  including  more  mission-relevant  secondary  tasks  or  increasing  the  number  of  robots 
controlled.  Including  more  secondary  tasks  could  reduce  the  amount  of  cognitive  resources 
available  to  be  allocated  to  both  primary  and  secondary  tasks,  in  turn  increasing  the  overall  effort 
required  to  maintain  performance  levels.  If  the  overall  effort  is  high,  the  introduction  of  operator 
aids  may  provide  more  benefit  to  performance  than  if  the  overall  effort  is  considered  low. 
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6.3  User  Preference 


The  user-preferences  data  addressing  the  preferences  hypothesis  (H4)  indicated  that  operators 
preferred  having  information  coming  from  one  or  more  aids  to  help  them  gain  a  complete 
understanding  of  the  situation.  Five  of  the  8  participants  showed  a  preference  for  conditions  that 
included  the  Travel  Planner.  Four  of  the  8  participants  preferred  conditions  with  the  Obstacle 
Map.  Interestingly,  5  of  the  8  participants  preferred  conditions  which  included  only  one  of  the 
operator  aids.  The  most  frequent  reason  for  this  preference  was  a  concern  about  clutter  on  the 
screen  and  the  potential  to  obscure  critical  information  from  the  user.  This  is  valuable  feedback 
that  reinforces  the  decision  made,  based  on  results  of  a  prior  study  (Evans  2013),  to  combine  the 
Short  Term  and  Long  Term  planners  into  the  single  Travel  Planner  aid  that  was  used  in  this 
current  study.  Additionally,  preference  data  add  another  source  of  information  when  considering 
future  adaptations  of  the  current  operator  aids.  An  example  of  this  would  be  to  provide  aids  with 
a  more  translucent,  see-through  presentation  so  as  to  minimize  the  occlusion  of  objects  present  in 
the  environment  behind  the  aids’  overlay  displays. 


7.  Conclusions 


These  study  results  suggest  that  information  about  robot  intent  can  be  inferred  through  the  use  of 
operator  aids.  The  operator  aids  allowed  the  participants  to  adjust  their  mental  model  of  what 
actions  the  robot  would  take  based  on  the  travel  plan,  obstacle  map,  and  rerouting  alerts 
displayed.  In  fact,  it  was  with  the  Travel  Planner  aid  that  operators  showed  the  most  consistent 
improvements  in  performance  across  the  measured  variables.  Knowing  what  actions  the  robot 
had  planned  may  have  allowed  the  participants  to  predict  both  the  robot’s  movements  and  the 
outcome  of  those  movements.  By  engaging  the  teleoperation  mode  less  frequently,  we  assume, 
the  operators  can  devote  a  larger  portion  of  their  attention  and  focus  on  the  primary  task  of  target 
detection.  However,  more  extensive  training  and  development  of  strategies  in  the  use  of  the  aids 
may  also  help  to  realize  improvements  in  performance. 

Another  related  area  of  interest  is  trust  (Hancock  et  al.  2011,  Freedy  et  al.  2007).  Operator  aids 
may  be  one  method  that  can  be  used  to  assist  Soldiers  in  appropriately  calibrating  trust  in  robot 
technology.  By  having  interfaces  that  build,  or  take  advantage  of,  shared  mental  models  within 
the  human-robot  team,  and  provide  transparency  into  the  reasoning  and  intent  of  the  robots, 
Soldiers  may  have  a  greater  understanding  of  the  robot  and  its  inner  workings.  It  is  posited  that  a 
greater  understanding  of  the  robot’s  reasoning  will  contribute  positively  to  trust  and  to 
performance — this  is  currently  an  active  area  of  research. 
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Appendix  A.  Examples  of  Ishihara  Color- Vision  Assessment 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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Color  Vision  Test 


What  numbers  do  you  see  revealed  in  the  patterns  of  dots  below?  Please  record 
the  number  on  the  answer  sheet  or,  if  you  do  not  see  a  number,  write  “NONE.” 


Question 


Question  2 


Question  5 


Question  6 


Please  turn  the  page  to  continue. . . 


26 


Appendix  B.  Example  of  Guilford-Zimmerman  Spatial  Visualization 

Assessment 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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The  Guilford-Zimmerman  Aptitude  Survey 


Part  6/Spatial  Visualization 


Name 


Date _  Score -  Sex:  M  F 


INSTRUCTIONS. 

This  is  a  test  of  how  well  you  are  able  to  visualize  spatial  position.  In  each  item  you  are  to  note  how  the  clock  would 
move  if  it  were  moved  as  indicated  by  the  arrow  on  the  sphere. 

Here  are  some  sample  items. 


A  B  C  D 


The  first  picture  at  the  left  shows  a  clock.  Next  to  it  is  a  sphere  with  an  arrow  marked  on  it.  The  arrow  shows  how  the 
clock  is  to  be  moved.  This  move  is  illustrated  (in  two  steps)  in  the  picture  below.  When  the  clock  is  moved  to  the  one- 
quarter  turn  shown  by  the  arrow,  it  is  then  in  position  B.  B  is  therefore  the  correct  answer.  You  would  record  this  by 
darkening  the  answer  space  right  below  B  on  your  answer  sheet.  (But  do  not  record  answers  to  sample  items.) 


Original 

Position 


Position  after  the  move 
has  been  completed 
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Appendix  C.  Example  of  Guilford-Zimmerman  Spatial  Orientation 

Assessment 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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The  Guilford-Zimmerman  Aptitude  Survey 


Part  5/Spatial  Orientation 


Name 


Date -  Score _  Sex:  M  F 


INSTRUCTIONS. 

This  is  a  test  of  your  ability  to  see  changes  in  direction  and  position.  In  each  item  you  are  to  note  how  the 
position  of  the  boat  has  changed  in  the  second  picture  from  the  original  position  in  the  first  picture. 


To  work  each  item:  First,  look  at  the  top  picture  and  see  where  the  motor  boat  is  headed.  Second,  look  at  the 
bottom  picture  and  note  the  CHANGE  in  the  boat's  heading.  Third,  mark  the  answer  that  shows  the  same  change  on 
the  separate  answer  sheet. 


Try  Sample  Item  2. 

This  also  shows  that  the  prow  of 
the  boat  is  to  the  right  of  the  aiming 
point.  So,  it  is  the  correct  answer.  - 


(If  the  boat  had  turned  to  the  left, 
instead  of  to  the  right,  the  correct 
answer  would  have  been  A.) 


This  is  the  aiming  point. 


This  is  the  same  aiming  point. 
The  motor  boat  is  now  headed  to 
the  right  of  it. 
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Appendix  D.  Example  of  NASA-TLX  Perceived  Workload  Assessment 


This  appendix  appears  in  its  original  form,  without  editorial  change. 


RATING  SCALE  DEFINITIONS 


TITLE 


ENDPOINTS  DESCRIPTIONS 


MENTAL  LOW/HIGH 

DEMAND 


PHYSICAL  LOW/HIGH 

DEMAND 


TEMPORAL  LOW/HIGH 

DEMAND 


PERFORMANCE  GOOD/POOR 


EFFORT  LOW/HIGH 


FRUSTRATION  LOW/HIGH 

LEVEL 


How  much  mental  and 
perceptual  activity  was  required 
(e.g.  thinking,  deciding,  calculating, 
remembering,  looking,  searching,  etc.)? 
was  the  task  easy  or  demanding,  simple 
or  complex,  exacting  or  forgiving? 


How  much  physical  activity  was 
required  (e.g.  pushing,  pulling, 
turning,  controlling,  activating, 
etc.)?  Was  the  task  easy  or 
demanding,  slow  or  brisk,  slack 
or  strenuous,  restful  or  laborious? 


How  much  time  pressure  did  you 
feel  due  to  the  rate  or  pace 
at  which  the  task  or  task  elements 
occurred?  was  the  pace  slow  and 
leisurely  or  rapid  and  frantic? 

How  successful  do  you  think  you 
were  in  accomplishing  the  goals 
of  the  task  set  by  the  experimenter 
(or  yourself)?  How  satisfied  were 
you  with  your  performance  in 
accomplishing  these  goals? 


How  hard  did  you  have  to  work 
(mentally  and  physically)  to 
accomplish  your  level  of 
performance? 


How  insecure,  discouraged,  irritated 
stressed,  and  annoyed  versus  secure, 
gratified,  content,  relaxed,  and 
complacent  did  you  feel  during  the  task? 
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Scoring  Form  1 


1.  Mental  Demand  -  Individual 


Low 

2.  Physical  Demand  -  individual 

I  I  I  I 

1 

1 

1 

1 

1 

High 

1 

Low 

3.  Temporal  Demand  -  Individual 

I  I  I  I 

1 

1 

1 

1 

1 

High 

1 

Low 

4.  Performance  -  Individual 

1 

1 

1 

1 

1 

High 

Good 

5.  Effort- Individual 

I  I  I  I 

1 

1 

1 

1 

1 

Bad 

l 

Low 

6.  Frustration  -  Individual 

1 

1 

1 

1 

1 

High 

Low 

High 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


ANCOVA 

ANOVA 

ANS 

ATO 

CAT 

COA 

GDRS 

GPS 

HRI 

IED 

MOUT 

OM 

RA 

s 

SA 

SOURCE 

3D 

TLX 

TP 

2D 

UGV 

WMI 


analysis  of  covariance 
analysis  of  variance 
Autonomous  Navigation  System 
Army  Technology  Objective 
Crew-integration  and  Automation  Test  bed 
course  of  action 

General  Dynamics  Robotics  System 
global  positioning  system 
human-robot  interaction 
improvised  explosive  device 
military  operations  in  urban  terrain 
Obstacle  Map 
Rerouting  Alert 
second 

situational  awareness 

Safe  Operations  for  Unmanned  Reconnaissance  in  Complex  Environments 

3 -dimensional 

Task  Load  Index 

Travel  Planner 

2-dimensional 

Unmanned  Ground  Vehicle 

Warfighter  Machine  Interface 
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